Automatic User Interface Data Generation

ABSTRACT

Techniques are disclosed relating to automatically synthesizing user interface (UI) component instances. In disclosed techniques a computer system receives a set of existing UI elements and a set of design rules for the set of existing elements, where design rules in the set of design rules indicate one or more allowed states for respective UI elements in the set of existing UI elements. The one or more allowed states may correspond to one or more visual characteristics. Using the set of existing UI elements, the computer system may then automatically generate a plurality of UI component instances based on the set of design rules, where a respective UI component instance includes a first UI element in a first allowed state. The computer system may then train, using the plurality of UI component instances, a machine learning model operable to automatically generate UI designs.

PRIORITY CLAIM

The present application is a continuation of U.S. application Ser. No.17/147,053, filed Jan. 12, 2021, which is incorporated by referenceherein in its entirety.

BACKGROUND Technical Field

Embodiments described herein relate to data processing, and morespecifically, to techniques for designing user interfaces, e.g., usingmachine learning.

Description of the Related Art

User interfaces are generally designed by multiple skilled designers anddevelopers to achieve both a functionality as well as an aestheticdesired by an entity employing the user interfaces. Designing customizeduser interfaces is time consuming and, thus, expensive. As such, variousentities may not have the necessary expertise or resources for obtainingcustomized user interface designs.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram illustrating example automatic generation of aplurality of UI component instances for use in training a machinelearning model, according to some embodiments.

FIGS. 2 and 3 are diagrams illustrating example component instances,according to some embodiments.

FIG. 4A is a diagram illustrating example allowed states for fourdifferent UI elements, according to some embodiments.

FIG. 4B is a block diagram illustrating an example tree structure builtfrom the UI elements shown in FIG. 4A, according to some embodiments.

FIG. 5A is a diagram illustrating example component instances generatedfrom three different UI elements with various different allowed states,according to some embodiments.

FIG. 5B is a diagram illustrating example component instances generatedfrom five different UI elements with various different allowed states,according to some embodiments.

FIG. 6 is a flow diagram illustrating a method for automaticallygenerating UI component instances from UI elements in various allowedstates for training a machine learning model to automatically designUIs, according to some embodiments.

FIG. 7 is a flow diagram illustrating a method for using a machinelearning model to automatically generate UI designs, according to someembodiments.

FIG. 8 is a block diagram illustrating an example computing device,according to some embodiments.

DETAILED DESCRIPTION

As discussed above, traditional techniques for generating userinterfaces (UIs) often involve tedious and expensive manual design by adeveloper who possesses both technical knowledge of programming as wellas a sense of style and aesthetic for the overall look of interfaces. Asa result, many entities opt for using a general template design withminimal changes or personalization when generating user interfaces e.g.,for their websites. This leaves little room for customization or theability for these websites to stand in contrast to websites of otherentities. In addition, such websites may quickly become outdated due tothese entities not having the funds to update their user interfaces.

In contrast, the disclosed techniques provide for automaticallygenerating user interfaces using machine learning techniques. In theSALESFORCE.COM context, however, training machine learning models todesign user interfaces automatically can be difficult due to a limitedavailability of existing user interface designs that follow desirabledesign requirements. Said another way, in some situations there is notenough training data (e.g., manually generated user interface designs)to sufficiently train a machine learning model. The disclosed techniquesare directed to automatically generating various UI component instancesfrom existing UI elements. These UI component instances are then usableto train a machine learning model to automatically generate customizedUI designs. In particular, rather than simply multiplying every possiblepermutation of different UI components together to determine a plethoraof UI designs, the disclosed techniques account for compatibility ofthese different UI components with one another as well as thecompatibility of the UI elements within these UI components. Forexample, the disclosed techniques define allowed states for various UIelements included in these components based on their compatibility withone another. In addition, the allowed states are defined based ondesirable visual characteristics. The allowed states are specified in aset of design rules associated with a given entity, which are used togenerate UI designs for that entity.

As used herein, the term “element” refers to the smallest granularity atwhich potentially-visible user interface items are specified. Individualimages, text strings, links, etc. are examples of user interfaceelements. Similarly, as used herein, the term “component” refers to aset of one or more (typically multiple) user interface elements. Forexample, a user interface component may be a button that includes a textstring and background, or may include several buttons. As used herein,the term “component instance” refers to a single combination of variouspossible permutations of a component generated from a set of one or moreelements in certain allowed states. For example, a button instance mightinclude a text element in the color red (one allowed state of this textelement) and a background element in the rectangular shape (one allowedstate of this background element).

As used herein, the term “design rules” refers to information specifyingvarious visual characteristics for UI elements including compatibilitybetween different UI elements. The design rules may specify, forexample, accessibility and stylistic requirements for various UIelements. For example, an accessibility requirement might be that awhite text element is not allowed to be rendered over a white backgroundelement within a given UI component. As one specific example, if abackground UI element has two states, white and gray, then according tothe design rules for the white text element, this text element has anallowed state (e.g., is displayable) in combination with the backgroundUI element only when the background element is in its gray state. Indisclosed techniques, a set of design rules dictate various allowedstates for UI elements. That is, the design rules lead to the allowedstates for various UI elements. These allowed states are then used inconjunction with an algorithm such as a script written in JavaScriptcode that is executable by various design programs (e.g., Sketch, React,Vue, Angular, etc.). This algorithm traverses a tree structure of UIelements in various allowed states (dictated by embedded design rules)to generate a plurality of UI component instances in the disclosedtechniques.

As used herein, the term “allowed state” refers to different possiblerenderings of a given user interface element. For example, a textelement may be displayed in the color red, green, or white. A textelement rendered in one of these three colors represents one of threepossible states in which this element can be rendered. As anotherexample, a box shape element might have two allowed states: a firststate in which it is displayed in the upper right corner of UIcomponents and a second state in which it is displayed in the lowerright corner of UI components. Examples of allowed states for UIelements are discussed in further detail below with reference to FIGS.4A-5B.

The disclosed techniques provide for automatic design of UIs byinserting various UI component instances into nodes of a tree structure.For example, each level of the tree structure represents another UIelement that is added to a UI component, while each node of the treerepresents a given UI component instance. The different branches at eachlevel of the tree indicate allowed states for the UI element added tothe UI component at that level. The disclosed tree structure is amechanism for predefining compatibility between various UI elements suchthat a plurality of UI component instances can automatically begenerated by traversing the tree structure.

The disclosed techniques for automatically generating UI componentinstances may advantageously reduce or remove the need for costly andtime-consuming manual design of these components. For example, manualgeneration of various UI component designs is a linear process that cantake hours, days, weeks, etc. The automatically generated UI componentdesigns are usable to train a machine learning model to automaticallydesign user interfaces. Further, the disclosed automatic UI componentgeneration techniques are exponentially faster than manual techniques atcreating a plurality of UI component designs usable to train a machinelearning model to generate UI designs. As one specific example, thedisclosed automated design techniques may be upwards of 30% faster thantraditional manual design techniques. The disclosed techniques providegreater efficiency gain over manual techniques as the complexity of UIcomponent designs increases. As such, the disclosed techniques allow forreal-time customization of UI designs for various entities withoutincurring large expenses.

Automatic UI Component Generation

FIG. 1 is a block diagram illustrating example automatic generation of aplurality of UI component instances for use in training a machinelearning model. In the illustrated embodiment, system 100 includes anautomatic UI generation system 170, and a computer system 110, which inturn includes a component generator module 160 and machine learningmodel 120.

Computer system 110, in the illustrated embodiment, receives a set 152of existing UI elements and a set 140 of design rules for the set ofexisting UI elements. In some embodiments, the set 152 of existing UIelements and the set 140 of design rules are received directly from auser interface designer. In other embodiments, computer system 110accesses a user interface database to retrieve the set 152 of existingUI elements and the set 140 of design rules. Existing UI elements in theset 152 may be any of various types of elements, including: text, link,image, label, etc.

After receiving this set 152 of elements, computer system 110 executesvia the tree module 130 of component generator module 160 an algorithmthat traverses a tree of the set 152 of existing UI elements accordingto the set 140 of design rules to automatically generate a plurality ofcomponent instances 162. For example, tree module 130 generates a treestructure using the set 152 of existing UI element. The levels of thetree represent respective ones of UI elements included in the set 152 ofexisting UI elements, the branches of the tree represent differentallowed states for the UI element corresponding to that level of thetree, and the leaf nodes represent component instances 162. An exampletree structure is discussed in detail below with reference to FIG. 4B.

In some embodiments, UI elements in the set 152 include design labelsassigned by designers based on design rules in the set 140. These designrules indicate allowed states for these UI elements. For example, afirst UI element may have an assigned label of “overlay.” This labelspecifies that when the particular element is displayed in the sameportion of a user interface as another UI element such that theyoverlap, this particular element must be displayed on top of the otherUI element. The overlay label may be assigned by a UI designer based onthe set 140 of design rules specifying that image elements are to bedisplayed in front of background elements, for example. As anotherexample, a designer may assign a label of “overlay, red” to a particularUI element included in the set 152, indicating that this element shouldbe displayed at the front of a UI component instance (i.e., on top ofother UI elements) in the color red.

In some embodiments, the set 140 of design rules specify accessibilityrequirements. For example, the design rules might specify whether one ormore UI elements are compatible with one another and can be combined ina UI component instance. As one specific example, a text element in thecolor white is not displayable over a white background element due tothe lack of visibility. The design rules may specify which colors arecompatible. In this specific example, the accessibility rules specifythat of the two states, white and red, only red is compatible with awhite background. The algorithm implemented by component generatormodule 160 might use this design rule to ensure that all generated UIcomponent instances with a white background only use the red text statefor the text element. In this example, the red text state is one allowedstate for this text element.

In some embodiments, the set of design rules specify style requirementsfor UI elements. The design rules might specify visual characteristicsof UI elements including one or more of the following: color, shape,size, opacity, orientation, font, position, etc. As one specificexample, a particular design rule might specify that the text of a“button” element is displayable in yellow, red, or blue. Based on thisrule, a UI designer might assign three different labels to this UIelement, producing three different allowed states for this element. Thatis, a first allowed state of the “button” text element is labeled“yellow,” a second allowed state is labeled “red,” and a third allowedstate is labeled “blue.”

In some embodiments, the set 140 of design rules includes predeterminedrules used by various designers when designing user interfaces. Forexample, the set 140 of design rules may be based on design bestpractices. These rules may specify when to use a divider line in a UI,how to group portions of a user interface together (to createcomponents), how to create a hierarchy in a UI design (e.g., the titletext of a webpage should be the largest element on the webpage, whichwould influence the allowed text size for other UI elements included inthe webpage), etc. In other embodiments, the set of rules may becustomized to a given entity based on the design criteria of thisentity. These rules may be referred to as branding guidelines. That is,a set of rules may be for a given set of UI elements belonging to aparticular entity, which are used to design UIs for that entity. Forexample, an entity may wish for their UI designs to follow a grayscalecolor palette and include shapes with sharp edges, providing their UIdesigns with a modern look. In this example, a developer may generate acustomized set of design rules for this entity corresponding to theirdesired design criteria.

In other embodiments, a given UI element or a given set of similar UIelements might have its own set of design rules. For example, a set ofborder elements used by ADIDAS on their website might have a set ofdesign rules specifying their color, shape, size, etc., while a set ofborder elements used by NIKE might have a different set of design rules.In various embodiments, accessibility rules, best practices, brandingguidelines, etc. specified in the set 140 of design rules are organizedinto a design hierarchy such that subsets of the set 140 of design rulesapply to different sets of similar UI elements.

Component generator module 160, in the illustrated embodiment, sends theplurality of UI component instances 162 to machine learning model 120for training. Example component instances are discussed in detail belowwith reference to FIGS. 2, 3, 5A, and 5B. These UI components may bereferred to as synthetic UI components. For example, the disclosedtechniques automatically synthesize new training data for machinelearning models in order to decrease costs associated with manualgeneration of UI data. In some embodiments, component generator module160 generates component instances 162 by assigning different UI elementsin the set 152 to various levels of a tree structure according to theirassigned labels. Once it has assigned the UI elements to differentlevels of the tree, component generator module 160 traverses the treestructure to determine the plurality of UI component instances 162. Thatis, the various leaf nodes of this tree structure represent respectivecomponent instances, while the branches of the tree indicate thedifferent allowed states of UI elements. An example tree structureutilized in generating a plurality of component instances is discussedin further detail below with reference to FIG. 4B.

Computer system 110 then uses these UI component instances 162 to traina machine learning model 120 to generate UI designs. Machine learningmodel 120 may be any of various types of machine learning modelsincluding: neural networks, logistic regression support vector machine,linear regression, random forests, etc. Computer system 110 provides atrained machine learning model 125 to automatic UI generation system170. Automatic UI generation system 170 then executes the trainedmachine learning model 125 to automatically generate various UI designsin real-time. In some embodiments, computer system 110 both trains andexecutes machine learning model 125. For example, computer system 110may perform training and may also automatically generate UI designsinstead of this action being performed by another system such as system170.

In other embodiments, trained machine learning model 125 is used toscore existing user interface designs. For example, model 125 may beused to score candidate UI designs from various different humandesigners in order to provide feedback to these designers. In oneparticular example, this feedback might specify how “on brand” their UIdesigns are for a given customer.

Note that various examples herein discuss automatic generation ofcompositions for user interfaces but these examples are discussed forpurposes of explanation and are not intended to limit the scope of thepresent disclosure. In other embodiments, the disclosed techniques areusable to generate designs for other mediums, such as movie posters,billboards, banner ads, etc.

In this disclosure, various “modules” operable to perform designatedfunctions are shown in the figures and described in detail above (e.g.,tree module 130, component generator module 160, etc.). As used herein,a “module” refers to software or hardware that is operable to perform aspecified set of operations. A module may refer to a set of softwareinstructions that are executable by a computer system to perform the setof operations. A module may also refer to hardware that is configured toperform the set of operations. A hardware module may constitutegeneral-purpose hardware as well as a non-transitory computer-readablemedium that stores program instructions, or specialized hardware such asa customized ASIC. Accordingly, a module that is described as being“executable” to perform operations refers to a software module, while amodule that is described as being “configured” to perform operationsrefers to a hardware module. A module that is described as “operable” toperform operations refers to a software module, a hardware module, orsome combination thereof. Further, for any discussion herein that refersto a module that is “executable” to perform certain operations, it is tobe understood that those operations may be implemented, in otherembodiments, by a hardware module “configured” to perform theoperations, and vice versa.

Example Component Instances

FIG. 2 is a diagram illustrating example component instances. In theillustrated embodiment, example 200 includes various component instances212. At the top of example 200, a button component is shown with onlythe text “button” and a link associated with this text. The text andlink elements 202 each have a single state. For example, the textelement is allowed to be rendered at the center of a component and inblack text. Similarly, the link element has a single state e.g., it isselectable by a user to navigate through a website.

Example 200 includes five different component instances 212. Forexample, at the top of FIG. 2, the text and link elements 202 are shownwithout a border as a first component instance. In contrast, componentinstance 204 includes two border elements: a first inner rectangularborder element surrounding the “button” text element and a second, outerbackground rectangular border element surrounding the inner rectangleborder. Component instance 206 includes a single oval shaped borderelement surrounding the “button” text. Component instance 208 includes acircular shaped element next to the “button” text. Finally, componentinstance 210 includes a rectangle border element surrounding the“button” text. Note that each of the shape elements (i.e., rectangles,ovals, circles) shown in the illustrated embodiment have a singleallowed state. For example, each of these shapes are shown with a whitebackground and at a particular location relative to the “button” textelement. In the illustrated embodiment, the rectangular border elementincluded in component instance 210 has a single allowed state:rectangular in shape and white in color.

FIG. 3 is a diagram illustrating example component instances. In theillustrated embodiment, example 300 includes two different componentinstances 302 and 304 composed of two UI elements. The star elementshown in this example is allowed two different states: overlayed andleft-aligned, or overlayed and right-aligned. For example, componentinstance 302 includes a rectangle element in its single allowed state(rendered behind the star element) and the star element in its firstallowed state (overlayed on the right side of the rectangle element). Incontrast, component instance 304 includes the rectangle element in itssingle allowed state and the star element in its second allowed state(overlayed on the left side of the rectangle element). That is, the starelement included in component instance 304 has been assigned labels“overlay” and “left-align,” while the rectangle element included incomponent instance 304 has been assigned a label “underlay.” As such,component generator module 160 knows to display the star element infront of and left-aligned with the rectangular element. In variousembodiments, design rules dictate the allowed states for various UIelements. In some situations, however, a designer may assign labels toUI elements according to other allowed states to indicate the specificsof these allowed states.

In various embodiments, the allowed states of UI elements are combinedto generate different component instances. For example, an outlineelement displayed with jagged edges might be combined with another,larger outline element that has smooth rectangular lines. Incombination, these two outline elements in their respective allowedstates make up a single component instance. Additional examples ofcombining different UI elements in various allowed states to generatecomponent instances are discussed in further detail below with referenceto FIGS. 5A and 5B.

Example Tree Structure

FIG. 4A is a diagram illustrating example allowed states for fourdifferent UI elements. In the illustrated embodiment, example 400includes various allowed states for four different UI elements 420 andexample design labels assigned to these UI elements, describing thecurrent state of these elements.

A first allowed state 430 is shown in the illustrated embodiment forelement 420A. One example design label 406A that a UI designer mightassign to element 420A is “black large box.” A first allowed state 432Ais shown for element 420B as a white box. Similarly, a second allowedstate 432B is shown for element 420B with an example design label 406Bof “gray medium box”. Two different allowed states 434A and 434B areshown for element 420C. The second allowed state 434B is assigned anexample design label 406C of “sunny small box.” Finally, element 420Dhas three different allowed states 436A, 436B, and 436C. Note that theselabels are examples and that any of various types of labelingconventions might employed by designers to describe a current allowedstate in which a UI element is to be combined with other UI elements.

FIG. 4B is a block diagram illustrating an example tree structure builtfrom the UI elements shown in FIG. 4A. In the illustrated embodiment,tree structure 404 includes various in-progress components 410A, 410B,410D, and 410E. Tree structure 404 also includes UI component instances410C, and 410F-410H. UI elements 420A-420D are shown in different onesof their allowed states. In the illustrated embodiment, the leaf nodesof tree structure 404 include component instances that are selected fortraining a machine learning model based on a set 140 of design rules.

In the illustrated embodiment, elements 420 are included in a givencomponent based on the set 140 of design rules, resulting in variouscomponent instances. The in-progress components shown in FIG. 4B areexamples of intermediate components resulting from the process ofgenerating viable component instances. For example, in-progresscomponents are not viable based on requirements for components specifiedin the set 140 of design rules. Rendering 402 shows the contents ofcomponent instance 410H when this instance is displayed in a userinterface. Rendering 402 is one example display of a component instancethat complies with a set 140 of design rules.

The tree structure 404 provides control over allowed permutations of acomponent without requiring a designer to explicitly design each viablepermutation (i.e., component instance). For example, the leaf nodes of atree structure 404 include component instances that are usable to traina machine learning model to automatically generate customized UIcomponents for various different uses. This tree structure mirrors thestructure of Hypertext Markup Language (HTML), providing for a smoothtransition from this design structure of component instances to UIprogram code (e.g., used to generate web pages).

In the illustrated embodiment, the root node of tree structure 404includes in-progress component 410A. Component 410A includes element420A in its single allowed state (e.g., a black box). Because element420A only has one allowed state it does not increase the number ofpermutations.

At the second level of tree 404, left and right branches fromin-progress component 410A are shown. These two branches include anin-progress component 410B and a component instance 410C. In theillustrated embodiment, a second element 420B has been added to thesetwo components 410B and 410C. Because element 420B has two differentallowed states, this results in the two different branches in the secondlevel of the tree 404. For example, in-progress component 410B includeselement 420A in black and element 420B in gray, while component instance410C includes element 420A in black and element 420B in white. That is,element 420B has two allowed states, gray or white, as shown in FIG. 4A.According to the set 140 of design rules, component instance 410C is aviable component even though it only includes two elements. That is, insome embodiments, leaf nodes included in the upper levels of a treestructure include viable component instances (according to the designrules).

At the third level of tree 404, two branches are shown from in-progresscomponent 410B. Specifically, at the third level of tree 404, a thirdelement 420C with two allowed states has been added. For example, thirdelement 420C can be shown as cloudy or sunny. As such, in-progresscomponent 410D includes element 420A shown in black, element 420B shownin gray, and element 420C shown as cloudy, while component 410E includeselement 420A shown in black, element 420B shown in gray, and element420C shown as sunny. Note that there are no branches off of componentinstance 410C due to element 420C not being allowed to combine withelement 420B when it is colored white (e.g., due to visibility issuesaccording to the accessibility rules included in the set 140 of designrules).

Finally, in the illustrated embodiment, a fourth level of tree 404 isshown with a fourth element 420D with three allowed states added togenerate UI component instances 410F, 410G, and 410H (which, in turn,includes two different versions). For example, the three allowed statesof element 420D are centered, left-aligned, or right-aligned. If,however, element 320C is shown as sunny (as in the case of in-progresscomponent 410E), then element 420D is only allowed to be right-alignedresulting in the second version of component instance 410H. For example,the set 140 of design rules dictate that element 420D can only becombined with element 420C in its sunny state, if element 420D isright-aligned.

As shown in the illustrated embodiment, in-progress component 410E onlyhas one branch to a second version of component instance 410H. The firstversion of component instance 410H includes element 420C shown ascloudy, while the second version of component instance 410H includeselement 420C shown as sunny. In particular, the second version ofcomponent instance 410H which branches from in-progress component 410Eincludes element 420A, element 420B in gray, element 420C shown as sunnyand element 420D right-aligned. In the illustrated embodiment, arendering 402 of the second version of component instance 410H is shown.

In the illustrated embodiment, in order to reach component instance410F, computer system 110 follows a path of allowed states for elementsthrough the tree structure. This path includes element 420A, element420B in gray, element 420C shown as cloudy, and element 420D showncentered. Similarly, the path to component instance 410G includeselement 420A, element 420B in gray, element 420C shown as cloudy, andelement 420D left-aligned.

Example UI Component Instances

FIG. 5A is a diagram illustrating example component instances generatedfrom three different UI elements with various different allowed states.In the illustrated embodiment, example 500 shows 18 different UIcomponent instances automatically generated using the disclosedtechniques from various allowed states of three different UI elements502A, 502B, and 502C.

UI element 502A, in the illustrated embodiment, is a background box withthree different allowed states: gray, white, and black. UI element 502Bis a weather icon with three allowed states: cloudy, sunny, and rainy.Finally, UI element 502C is a small green square (the color greenrepresented by shading) that has two allowed states: left-aligned andright-aligned. The component permutations generated from these threeelements with their eight (total) allowed states result in the 18different UI component instances shown in FIG. 5A.

FIG. 5B is a diagram illustrating example component instances generatedfrom five different UI elements with various different allowed states.In the illustrated embodiment, example 502 shows a more complex versionof example 500. Specifically, example 502 illustrates 63 UI componentinstances generated from five different UI elements 512A-512E withvarious different allowed states. In the illustrated embodiment, example502 includes all possible UI component instances that might be generatedfrom the five different UI elements in their different allowed states.

In the illustrated embodiment, UI element 512A (the outer squarebackground) has three allowed states (gray, white, and black), UIelement 512B (the middle square) has three allowed states (gray, white,and black), UI element 512C (the weather icon) has three allowed states(cloudy, sunny, and rainy), UI element 512D (the small center square)has three allowed states (red, white, and purple), and UI element 512E(the right-aligned small square) has two allowed states (green andpurple). Note that the lighter dotted shading on element 512E representsthe color green, the darker dotted shading on element 512D representsthe color red, and the checkerboard shading that colors either element512D and/or element 512E in different ones of their allowed statesrepresents the color purple in FIG. 5B. In total, the five different UIelements 512 include fourteen different allowed states. According to theset 140 of design rules, these five UI elements 512 with their fourteendifferent allowable states can be combined to generate the 63 differentpermutations (UI component instances) shown in FIG. 5B.

Example Methods

FIG. 6 is a flow diagram illustrating a method 600 for automaticallygenerating UI component instances from UI elements in various allowedstates for training a machine learning model to automatically designUIs, according to some embodiments. The method shown in FIG. 6 may beused in conjunction with any of the computer circuitry, systems,devices, elements, or components disclosed herein, among other devices.In various embodiments, some of the method elements shown may beperformed concurrently, in a different order than shown, or may beomitted. Additional method elements may also be performed as desired.

At element 610, in the illustrated embodiment, a computer systemreceives a set of existing user interface (UI) elements and a set ofdesign rules for the set of existing UI elements, where design rules inthe set of design rules indicate one or more allowed states forrespective UI elements in the set of existing UI elements. In someembodiments, the one or more allowed states correspond to one or morevisual characteristics. In some embodiments, the set of design rulesspecify, for UI elements in the set of existing UI elements,compatibility between the UI elements and visual characteristics of theUI elements. For example, the design rules might specify that a whitecircle is not displayable over (i.e., compatible with) a whitebackground. In some embodiments, visual characteristics include: color,shape, size, opacity, font, orientation, and position.

In some embodiments, the automatically generating includes generating,based on the one or more allowed states for respective UI elements inthe set of existing UI elements, a tree structure, where levels of thetree structure correspond to respective UI elements. In someembodiments, branches of the tree structure at a respective levelcorrespond to different allowed states of UI element corresponding tothat respective level. The tree structure discussed above and shown inFIG. 4B is one example of the tree structure generated based on existingUI elements in their allowed states. In some embodiments, generating thetree structure includes traversing the tree structure to identify theplurality of UI component instances, where leaf nodes of the treestructure represent different ones of the plurality of UI componentinstances.

At 620, the computer system automatically generates, using the set ofexisting UI elements, a plurality of UI component instances, where arespective UI component instance includes a first UI element in a firstallowed state. In some embodiments, an allowed state for the first UIelement is conditionally based on an allowed state for a second,different UI element. For example, an allowed color of the text “button”is determined based on the color of the background of this button.Specifically, in this example, the black “button” text is not allowed tobe displayed on a black background. In some embodiments, the respectiveUI component instance includes the first UI element in a second allowedstate and a second UI element in a first allowed state. In someembodiments, the set of design rules specify that the first allowedstate of the first UI element includes a first color and that the secondallowed state of the first UI element includes a second, differentcolor. For example, the respective UI component might include a smallyellow star (the first element in a first state) displayed over a bluebox (the second element in a second state) and a black border (a thirdelement in a first allowed state).

At 630, the computer system trains, using the plurality of UI componentinstances, a machine learning model operable to automatically generateUI designs. In some embodiments, the computer system automaticallygenerates, based on the plurality of UI component instances, a pluralityof UI designs. For example, a UI design might be a webpage of a web sitethat includes a combination of different elements and components, whilecomponents within the webpage include various different UI elements. Insome embodiments, the computer system trains, using the plurality of UIdesign instances, the machine learning model trained using the pluralityof UI component instances. For example, instead of, or in addition to,training a machine learning model using the UI component instancesgenerated at element 620, the computer system might generate UI designsfrom the UI component instances using similar techniques to those usedto generate the component instances. The computer system then uses theseUI design instances to train a machine learning model instead of (or inaddition to) using component instances for the training. In someembodiments, automatically generating the plurality of UI designinstances includes assigning ones of the plurality of UI componentinstances to respective levels of a tree structure and traversing thetree structure to determine UI design instances. Using a tree structureto determine various UI design instances takes into accountcompatibility between various UI components included in the UI designs.For example, a button component with black text and a pink backgroundmay not be compatible with (e.g., displayable within) a list componentthat is also pink.

FIG. 7 is a flow diagram illustrating a method 700 for using a machinelearning model to automatically generate UI designs, according to someembodiments. The method shown in FIG. 7 may be used in conjunction withany of the computer circuitry, systems, devices, elements, or componentsdisclosed herein, among other devices. In various embodiments, some ofthe method elements shown may be performed concurrently, in a differentorder than shown, or may be omitted. Additional method elements may alsobe performed as desired.

At 710, in the illustrated embodiment, a computing system automaticallygenerates, using a machine learning model trained using a plurality ofsynthetic UI component instances, one or more user interface (UI)designs. For example, the machine learning model might be used tocustomize a webpage for a particular user in real-time based on thingsthat the particular user clicks on. A webpage is one example of a UIdesign that computing system might automatically generate from theplurality of UI component instances that are automatically generated atelement 620 above.

At 720, in order to obtain the trained machine learning model used atelement 710, the computing system receives a set of existing userinterface (UI) elements and a set of design rules for the set ofexisting UI elements where the design rules indicate one or more allowedstates for respective UI elements in the set of existing UI elements. Insome embodiments, the one or more allowed states correspond to one ormore visual characteristics. For example, an allowed state for a UIelement might correspond to the location of that element within aparticular UI component instance.

At 730, further in order to obtain the trained machine learning modelused at element 710, the computing system automatically generates, usingthe set of existing UI elements, the plurality of synthetic UI componentinstances, where a respective UI component instance includes a first UIelement in a first allowed state. In some embodiments, the automaticallygenerating is performed based on the set of design rules. In someembodiments, the receiving and automatically generating are performed bya system other than the computing system. For example, a first systemmight automatically synthesize UI component instances and train amachine learning model using these instances, while a second, differentsystem uses the trained machine learning model to automatically generateUI designs.

Example Computing Device

Turning now to FIG. 8, a block diagram of one embodiment of computingdevice (which may also be referred to as a computing system) 810 isdepicted. Computing device 810 may be used to implement various portionsof this disclosure. Computing device 810 may be any suitable type ofdevice, including, but not limited to, a personal computer system,desktop computer, laptop or notebook computer, mainframe computersystem, web server, workstation, or network computer. As shown,computing device 810 includes processing unit 850, storage 812, andinput/output (I/O) interface 830 coupled via an interconnect 860 (e.g.,a system bus). I/O interface 830 may be coupled to one or more I/Odevices 840. Computing device 810 further includes network interface832, which may be coupled to network 820 for communications with, forexample, other computing devices.

In various embodiments, processing unit 850 includes one or moreprocessors. In some embodiments, processing unit 850 includes one ormore coprocessor units. In some embodiments, multiple instances ofprocessing unit 850 may be coupled to interconnect 860. Processing unit850 (or each processor within 850) may contain a cache or other form ofon-board memory. In some embodiments, processing unit 850 may beimplemented as a general-purpose processing unit, and in otherembodiments it may be implemented as a special purpose processing unit(e.g., an ASIC). In general, computing device 810 is not limited to anyparticular type of processing unit or processor subsystem.

Storage subsystem 812 is usable by processing unit 850 (e.g., to storeinstructions executable by and data used by processing unit 850).Storage subsystem 812 may be implemented by any suitable type ofphysical memory media, including hard disk storage, floppy disk storage,removable disk storage, flash memory, random access memory (RAM-SRAM,EDO RAM, SDRAM, DDR SDRAM, RDRAM, etc.), ROM (PROM, EEPROM, etc.), andso on. Storage subsystem 812 may consist solely of volatile memory, inone embodiment. Storage subsystem 812 may store program instructionsexecutable by computing device 810 using processing unit 850, includingprogram instructions executable to cause computing device 810 toimplement the various techniques disclosed herein.

I/O interface 830 may represent one or more interfaces and may be any ofvarious types of interfaces configured to couple to and communicate withother devices, according to various embodiments. In one embodiment, I/Ointerface 830 is a bridge chip from a front-side to one or moreback-side buses. I/O interface 830 may be coupled to one or more I/Odevices 840 via one or more corresponding buses or other interfaces.Examples of I/O devices include storage devices (hard disk, opticaldrive, removable flash drive, storage array, SAN, or an associatedcontroller), network interface devices, user interface devices or otherdevices (e.g., graphics, sound, etc.).

Various articles of manufacture that store instructions (and,optionally, data) executable by a computing system to implementtechniques disclosed herein are also contemplated. The computing systemmay execute the instructions using one or more processing elements. Thearticles of manufacture include non-transitory computer-readable memorymedia. The contemplated non-transitory computer-readable memory mediainclude portions of a memory subsystem of a computing device as well asstorage media or memory media such as magnetic media (e.g., disk) oroptical media (e.g., CD, DVD, and related technologies, etc.). Thenon-transitory computer-readable media may be either volatile ornonvolatile memory.

The present disclosure includes references to “embodiments,” which arenon-limiting implementations of the disclosed concepts. References to“an embodiment,” “one embodiment,” “a particular embodiment,” “someembodiments,” “various embodiments,” and the like do not necessarilyrefer to the same embodiment. A large number of possible embodiments arecontemplated, including specific embodiments described in detail, aswell as modifications or alternatives that fall within the spirit orscope of the disclosure. Not all embodiments will necessarily manifestany or all of the potential advantages described herein.

Unless stated otherwise, the specific embodiments are not intended tolimit the scope of claims that are drafted based on this disclosure tothe disclosed forms, even where only a single example is described withrespect to a particular feature. The disclosed embodiments are thusintended to be illustrative rather than restrictive, absent anystatements to the contrary. The application is intended to cover suchalternatives, modifications, and equivalents that would be apparent to aperson skilled in the art having the benefit of this disclosure.

Particular features, structures, or characteristics may be combined inany suitable manner consistent with this disclosure. The disclosure isthus intended to include any feature or combination of featuresdisclosed herein (either explicitly or implicitly), or anygeneralization thereof. Accordingly, new claims may be formulated duringprosecution of this application (or an application claiming prioritythereto) to any such combination of features. In particular, withreference to the appended claims, features from dependent claims may becombined with those of the independent claims and features fromrespective independent claims may be combined in any appropriate mannerand not merely in the specific combinations enumerated in the appendedclaims.

For example, while the appended dependent claims are drafted such thateach depends on a single other claim, additional dependencies are alsocontemplated, including the following: Claim 4 (could depend from any ofclaims 1-3); claim 5 (any preceding claim); claim 6 (claim 4), etc.Where appropriate, it is also contemplated that claims drafted in onestatutory type (e.g., apparatus) suggest corresponding claims of anotherstatutory type (e.g., method).

Because this disclosure is a legal document, various terms and phrasesmay be subject to administrative and judicial interpretation. Publicnotice is hereby given that the following paragraphs, as well asdefinitions provided throughout the disclosure, are to be used indetermining how to interpret claims that are drafted based on thisdisclosure.

References to the singular forms such “a,” “an,” and “the” are intendedto mean “one or more” unless the context clearly dictates otherwise.Reference to “an item” in a claim thus does not preclude additionalinstances of the item.

The word “may” is used herein in a permissive sense (i.e., having thepotential to, being able to) and not in a mandatory sense (i.e., must).

The terms “comprising” and “including,” and forms thereof, areopen-ended and mean “including, but not limited to.”

When the term “or” is used in this disclosure with respect to a list ofoptions, it will generally be understood to be used in the inclusivesense unless the context provides otherwise. Thus, a recitation of “x ory” is equivalent to “x or y, or both,” covering x but not y, y but notx, and both x and y. On the hand, a phrase such as “either x or y, butnot both” makes clear that “or” is being used in the exclusive sense.

A recitation of “w, x, y, or z, or any combination thereof” or “at leastone of . . . w, x, y, and z” is intended to cover all possibilitiesinvolving a single element up to the total number of elements in theset. For example, given the set [w, x, y, z], these phrasings cover anysingle element of the set (e.g., w but not x, y, or z), any two elements(e.g., w and x, but not y or z), any three elements (e.g., w, x, and y,but not z), and all four elements. The phrase “at least one of . . . w,x, y, and z” thus refers to at least one of element of the set [w, x, y,z], thereby covering all possible combinations in this list of options.This phrase is not to be interpreted to require that there is at leastone instance of w, at least one instance of x, at least one instance ofy, and at least one instance of z.

Various “labels” may proceed nouns in this disclosure. Unless contextprovides otherwise, different labels used for a feature (e.g., “firstcircuit,” “second circuit,” “particular circuit,” “given circuit,” etc.)refer to different instances of the feature. The labels “first,”“second,” and “third” when applied to a particular feature do not implyany type of ordering (e.g., spatial, temporal, logical, etc.), unlessstated otherwise.

Within this disclosure, different entities (which may variously bereferred to as “units,” “circuits,” other components, etc.) may bedescribed or claimed as “configured” to perform one or more tasks oroperations. This formulation—[entity] configured to [perform one or moretasks]—is used herein to refer to structure (i.e., something physical).More specifically, this formulation is used to indicate that thisstructure is arranged to perform the one or more tasks during operation.A structure can be said to be “configured to” perform some task even ifthe structure is not currently being operated. Thus, an entity describedor recited as “configured to” perform some task refers to somethingphysical, such as a device, circuit, memory storing program instructionsexecutable to implement the task, etc. This phrase is not used herein torefer to something intangible.

The term “configured to” is not intended to mean “configurable to.” Anunprogrammed FPGA, for example, would not be considered to be“configured to” perform some specific function. This unprogrammed FPGAmay be “configurable to” perform that function, however.

Reciting in the appended claims that a structure is “configured to”perform one or more tasks is expressly intended not to invoke 35 U.S.C.§ 112(f) for that claim element. Should Applicant wish to invoke Section112(f) during prosecution, it will recite claim elements using the“means for” [performing a function] construct.

The phrase “based on” is used to describe one or more factors thataffect a determination. This term does not foreclose the possibilitythat additional factors may affect the determination. That is, adetermination may be solely based on specified factors or based on thespecified factors as well as other, unspecified factors. Consider thephrase “determine A based on B.” This phrase specifies that B is afactor that is used to determine A or that affects the determination ofA. This phrase does not foreclose that the determination of A may alsobe based on some other factor, such as C. This phrase is also intendedto cover an embodiment in which A is determined based solely on B. Asused herein, the phrase “based on” is synonymous with the phrase “basedat least in part on.”

The phrase “in response to” describes one or more factors that triggeran effect. This phrase does not foreclose the possibility thatadditional factors may affect or otherwise trigger the effect. That is,an effect may be solely in response to those factors, or may be inresponse to the specified factors as well as other, unspecified factors.Consider the phrase “perform A in response to B.” This phrase specifiesthat B is a factor that triggers the performance of A. This phrase doesnot foreclose that performing A may also be in response to some otherfactor, such as C. This phrase is also intended to cover an embodimentin which A is performed solely in response to B.

1-20. (canceled)
 21. A method, comprising: automatically generating, bya computer system based on a set of existing user interface (UI)elements, a plurality of UI component instances, wherein theautomatically generating is performed by traversing a tree structureaccording to one or more allowed states for UI elements included in theset of existing UI elements to identify the plurality of UI componentinstances; and training, by the computer system using the plurality ofUI component instances, a machine learning model to automaticallygenerate UI designs.
 22. The method of claim 21, wherein the one or moreallowed states correspond to one or more visual characteristics, andwhere the one or more allowed states are indicated by a set of designrules for the existing set of UI elements.
 23. The method of claim 22,wherein the set of design rules specify compatibility between UIelements in the set of existing UI elements.
 24. The method of claim 22,wherein the set of design rules specify that a first allowed state of afirst UI element includes a first shape and that a second allowed stateof the first UI element includes a second, different shape.
 25. Themethod of claim 21, wherein the trained machine learning model isoperable to score one or more candidate UI designs.
 26. The method ofclaim 21, wherein at least one component instance in the plurality of UIcomponent instances includes a first UI element in a first allowedstate.
 27. The method of claim 26, wherein the at least one componentinstance further includes the first UI element in a second allowed stateand a second UI element in a first allowed state.
 28. The method ofclaim 21, wherein the leaf nodes of the tree structure representdifferent ones of the plurality of UI component instances.
 29. Anon-transitory computer-readable medium having instructions storedthereon that are executable by a computer system to perform operationscomprising: automatically generate, based on a set of existing userinterface (UI) elements, a plurality of UI component instances, whereinthe automatically generating is performed by traversing a tree structureaccording to one or more allowed states for UI elements included in theset of existing UI elements to identify the plurality of UI componentinstances; and training, using the plurality of UI component instances,a machine learning model to automatically generate UI designs.
 30. Thenon-transitory computer-readable medium of claim 29, wherein the one ormore allowed states correspond to one or more visual characteristics,and where the one or more allowed states are indicated by a set ofdesign rules for the existing set of UI elements.
 31. The non-transitorycomputer-readable medium of claim 30, wherein the set of design rulesspecify compatibility between UI elements in the set of existing UIelements.
 32. The non-transitory computer-readable medium of claim 29,wherein at least one component instance in the plurality of UI componentinstances includes a first UI element in a first allowed state.
 33. Thenon-transitory computer-readable medium of claim 32, wherein the atleast one component instance further includes the first UI element in asecond allowed state and a second UI element in a first allowed state.34. The non-transitory computer-readable medium of claim 33, wherein theleaf nodes of the tree structure represent different ones of theplurality of UI component instances.
 35. A method, comprising:automatically generating, by a computing system using a machine learningmodel trained using a plurality of synthetic UI component instances, oneor more user interface (UI) designs, wherein the plurality of syntheticUI component instances are automatically generated by: automaticallygenerating, by a computer system based on a set of existing userinterface (UI) elements, a plurality of UI component instances, whereinthe automatically generating is performed by traversing a tree structureaccording to one or more allowed states for UI elements included in theset of existing UI elements to identify the plurality of UI componentinstances; and training, by the computer system using the plurality ofUI component instances, a machine learning model to automaticallygenerate UI designs.
 36. The method of claim 35, wherein the one or moreallowed states correspond to one or more visual characteristics, andwhere the one or more allowed states are indicated by a set of designrules for the existing set of UI elements.
 37. The method of claim 36,wherein the set of design rules specify compatibility between UIelements in the set of existing UI elements.
 38. The method of claim 36,wherein the set of design rules specify one or more visualcharacteristics of the following visual characteristics: color, shape,size, opacity, font, orientation, and position.
 39. The method of claim35, wherein at least one component instance in the plurality of UIcomponent instances includes a first UI element in a first allowedstate.
 40. The method of claim 35, wherein the trained machine learningmodel is operable to score one or more candidate UI designs.